{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 数据库下载\n",
    "\n",
    "笔记本各单元格是阻断式的,只能一个一个执行.\n",
    "\n",
    "下列的wget命令若在终端直接执行,可以看到下载速度和进度,且可以同时下载."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "import os\n",
    "import subprocess\n",
    "import pandas as pd\n",
    "import collections\n",
    "import xml.etree.ElementTree as ET"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "切换下载目录:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 保存到你想存放数据库文件的位置\n",
    "data = \"/home/regen/projects/cpie_dbs\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unix系统"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stdout: BindingDB.csv\n",
      "ChEMBL.csv\n",
      "CTD.csv\n",
      "DB.csv\n",
      "DrugCentral.csv\n",
      "DTC.csv\n",
      "STITCH.tsv\n",
      "\n",
      "stderr: \n"
     ]
    }
   ],
   "source": [
    "commands = f\"ls {data}\"\n",
    "\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "print('stdout:', result.stdout)\n",
    "print('stderr:', result.stderr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BindingDB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BDB 一般每月更新数据库.\n",
    "\n",
    "如果下面的命令返回404状态,手动修改月份为几个月前.\n",
    "\n",
    "下载约10min."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = datetime.now().strftime('%m')\n",
    "y = datetime.now().strftime('%Y') \n",
    "\n",
    "commands = f\"wget -P {data}/ https://www.bindingdb.org/bind/downloads/BindingDB_All_{y}{m}_tsv.zip -O {data}/BindingDB.tsv.zip && \\\n",
    "            unzip {data}/BindingDB.tsv.zip -d {data}/ && rm {data}/BindingDB.tsv.zip && \\\n",
    "            mv {data}/BindingDB*.tsv {data}/BindingDB.tsv\"\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, text=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ChEMBL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ChEMBL需要在[网页](https://www.ebi.ac.uk/chembl/web_components/explore/activities/)等待`.csv`文件的生成\n",
    "\n",
    "- 点击左侧的*target organism -> Homo sapiens*类别\n",
    "- 点击页面上方的csv下载按钮,等待生成csv文件(约45min)\n",
    "- 点击下载按钮(约10min,下载失败可重试,在断点重连)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Filtering section button for Homo sapiens](ChEMBL-filtering.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![CSV button to generate annotations file](ChEMBL-csv.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Button to download annotations file](ChEMBL-download.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下载的zip文件中包含多个csv文件,需要进行合并:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "解压文件."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  /home/regen/projects/cpie_dbs/ChEMBL.zip\n",
      "  inflating: /home/regen/projects/cpie_dbs/chembl/DOWNLOAD-Dk2SYbS9_2ICqo5pvFaYNnuGdjLjIb67gBAv_nvWflo=.csv  \n",
      "  inflating: /home/regen/projects/cpie_dbs/chembl/DOWNLOAD-Dk2SYbS9_2ICqo5pvFaYNnuGdjLjIb67gBAv_nvWflo=_part2.csv  \n",
      "  inflating: /home/regen/projects/cpie_dbs/chembl/DOWNLOAD-Dk2SYbS9_2ICqo5pvFaYNnuGdjLjIb67gBAv_nvWflo=_part3.csv  \n",
      "  inflating: /home/regen/projects/cpie_dbs/chembl/DOWNLOAD-Dk2SYbS9_2ICqo5pvFaYNnuGdjLjIb67gBAv_nvWflo=_part4.csv  \n",
      "  inflating: /home/regen/projects/cpie_dbs/chembl/DOWNLOAD-Dk2SYbS9_2ICqo5pvFaYNnuGdjLjIb67gBAv_nvWflo=_part5.csv  \n",
      "  inflating: /home/regen/projects/cpie_dbs/chembl/DOWNLOAD-Dk2SYbS9_2ICqo5pvFaYNnuGdjLjIb67gBAv_nvWflo=_part6.csv  \n",
      "  inflating: /home/regen/projects/cpie_dbs/chembl/DOWNLOAD-Dk2SYbS9_2ICqo5pvFaYNnuGdjLjIb67gBAv_nvWflo=_part7.csv  \n",
      "  inflating: /home/regen/projects/cpie_dbs/chembl/DOWNLOAD-Dk2SYbS9_2ICqo5pvFaYNnuGdjLjIb67gBAv_nvWflo=_part8.csv  \n"
     ]
    }
   ],
   "source": [
    "commands = f\"mkdir -p {data}/chembl && \\\n",
    "            unzip {data}/ChEMBL.zip -d {data}/chembl && \\\n",
    "            rm {data}/ChEMBL.zip\"\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, text=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "合并csv文件(需要约40GB内存):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def merge_csv_files(folder_path, output_file):\n",
    "    \n",
    "    files = os.listdir(folder_path)\n",
    "    \n",
    "    csv_files = [file for file in files if file.endswith('.csv')]\n",
    "    \n",
    "    if len(csv_files) == 0:\n",
    "        print(\"No CSV files found in the folder.\")\n",
    "        return\n",
    "    \n",
    "    merged_df = pd.DataFrame()\n",
    "    \n",
    "    for file in csv_files:\n",
    "        file_path = os.path.join(folder_path, file)\n",
    "        df = pd.read_csv(file_path,sep=';',low_memory=False)\n",
    "        merged_df = pd.concat([merged_df, df], ignore_index=True)\n",
    "    \n",
    "    merged_df.to_csv(output_file, index=False)\n",
    "\n",
    "folder_path = f'{data}/chembl/'\n",
    "output_file = f'{data}/ChEMBL_merge.csv' # 为避免内存溢出直接覆盖文件可能损坏,保存到另一文件\n",
    "merge_csv_files(folder_path, output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "删除下载的文件:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "commands = f\"rm -r {data}/chembl/ && mv {data}/ChEMBL_merge.csv {data}/ChEMBL.csv\"\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, text=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CTD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stdout: \n"
     ]
    }
   ],
   "source": [
    "commands = f\"wget https://ctdbase.org/reports/CTD_chem_gene_ixns.csv.gz -O {data}/CTD.csv.gz\"\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)\n",
    "\n",
    "# Extract csv using pandas\n",
    "ctd = pd.read_csv(f\"{data}/CTD.csv.gz\", compression='gzip', on_bad_lines='skip', skiprows=28)\n",
    "ctd.to_csv(f\"{data}/CTD.csv\", sep=',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "移除下载数据:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stdout: \n"
     ]
    }
   ],
   "source": [
    "commands = f\"rm {data}/CTD.csv.gz\"\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Drugbank"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DrugBank需要学术协议来下载,访问[注册页](https://go.drugbank.com/public_users/sign_up)获得下载许可.\n",
    "\n",
    "然后访问[下载页面](https://go.drugbank.com/releases/latest)手动下载或者运行以下单元格下载(更改注册邮箱和密码)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![DrugBank complete database download](DrugBank.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下载压缩包:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "email = 'YOURUSERNAME' # 自行更改\n",
    "password = 'YOURPASSWORD'# 自行更改\n",
    "\n",
    "commands = f\"wget --user={email} --password={password} https://go.drugbank.com/releases/latest/downloads/all-full-database -O {data}/drugbank_all_full_database.xml.zip\"\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "解压文件:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "commands = f\"unzip {data}/drugbank_all_full_database.xml.zip -d {data}/ && \\\n",
    "            rm {data}/drugbank_all_full_database.xml.zip\"\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "转化为csv文件:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def collapse_list_values(row):\n",
    "    for key, value in row.items():\n",
    "        if isinstance(value, list):\n",
    "            row[key] = '|'.join(value)\n",
    "    return row\n",
    "\n",
    "def xml2csv(file_path, output_file):\n",
    "    tree = ET.parse(file_path)\n",
    "    root = tree.getroot()\n",
    "\n",
    "    ns = '{http://www.drugbank.ca}'\n",
    "    inchikey_template = \"{ns}calculated-properties/{ns}property[{ns}kind='InChIKey']/{ns}value\"\n",
    "    inchi_template = \"{ns}calculated-properties/{ns}property[{ns}kind='InChI']/{ns}value\"\n",
    "\n",
    "    rows = list()\n",
    "    for i, drug in enumerate(root):\n",
    "        row = collections.OrderedDict()\n",
    "        assert drug.tag == ns + 'drug'\n",
    "        row['type'] = drug.get('type')\n",
    "        row['drugbank-id'] = drug.findtext(ns + \"drugbank-id[@primary='true']\")\n",
    "        row['name'] = drug.findtext(ns + \"name\")\n",
    "        row['description'] = drug.findtext(ns + \"description\")\n",
    "        row['InChIKey'] = drug.findtext(inchikey_template.format(ns = ns))\n",
    "        rows.append(row)\n",
    "\n",
    "    rows = list(map(collapse_list_values, rows))\n",
    "\n",
    "    columns = ['drugbank-id', 'name', 'type', 'InChIKey', 'description']\n",
    "    drugbank_df = pd.DataFrame.from_dict(rows)[columns]\n",
    "\n",
    "    protein_rows = list()\n",
    "    for i, drug in enumerate(root):\n",
    "        drugbank_id = drug.findtext(ns + \"drugbank-id[@primary='true']\")\n",
    "        for category in ['target', 'enzyme', 'carrier', 'transporter']:\n",
    "            proteins = drug.findall('{ns}{cat}s/{ns}{cat}'.format(ns=ns, cat=category))\n",
    "            for protein in proteins:\n",
    "                row = {'drugbank-id': drugbank_id, 'protein_type': category}\n",
    "                row['protein_name'] = protein.findtext('{}name'.format(ns))\n",
    "                row['organism'] = protein.findtext('{}organism'.format(ns))\n",
    "                actions = protein.findall('{ns}actions/{ns}action'.format(ns=ns))\n",
    "                row['actions'] = '|'.join(action.text for action in actions)\n",
    "                uniprot_ids = [polypep.text for polypep in protein.findall(\n",
    "                    \"{ns}polypeptide/{ns}external-identifiers/{ns}external-identifier[{ns}resource='UniProtKB']/{ns}identifier\".format(ns=ns))]            \n",
    "                if len(uniprot_ids) == 1:\n",
    "                    row['uniprot_id'] = uniprot_ids[0]\n",
    "                hgnc_ids = [polypep.text for polypep in protein.findall(\n",
    "                    \"{ns}polypeptide/{ns}external-identifiers/{ns}external-identifier[{ns}resource='HUGO Gene Nomenclature Committee (HGNC)']/{ns}identifier\".format(ns=ns))]            \n",
    "                if len(hgnc_ids) == 1:\n",
    "                    row['HGNC'] = hgnc_ids[0]\n",
    "                protein_rows.append(row)\n",
    "\n",
    "    protein_df = pd.DataFrame.from_dict(protein_rows)\n",
    "\n",
    "    drugbank = pd.merge(drugbank_df, protein_df, on='drugbank-id', how='left')\n",
    "\n",
    "    drugbank.to_csv(output_file, sep=',', index=False)\n",
    "\n",
    "file_path = f'{data}/full database.xml'\n",
    "output_file = f'{data}/DB.csv'\n",
    "xml2csv(file_path, output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "移除下载文件:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "commands = f'rm \"{data}/full database.xml\"'\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DrugCentral\n",
    "\n",
    "本笔记提供的链接目前是最新版本,与cpiextract仓库中下载的版本一致,可跳过."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stdout: \n"
     ]
    }
   ],
   "source": [
    "commands = f\"wget https://unmtid-dbs.net/download/DrugCentral/2021_09_01/drug.target.interaction.tsv.gz -O {data}/DC.tsv.gz && \\\n",
    "            gzip -d {data}/DC.tsv.gz && \\\n",
    "            wget https://unmtid-dbs.net/download/DrugCentral/2021_09_01/structures.smiles.tsv -O {data}/DC_comps.tsv\"\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "合并文件:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "DC = pd.read_csv(f'{data}/DC.tsv', sep='\\t')\n",
    "DC_comps = pd.read_csv(f'{data}/DC_comps.tsv', sep='\\t')\n",
    "\n",
    "DC_comps = DC_comps[['ID', 'SMILES', 'InChI', 'InChIKey']].rename(columns={'ID':'STRUCT_ID'})\n",
    "\n",
    "DC = pd.merge(DC, DC_comps, on='STRUCT_ID', how='left')\n",
    "\n",
    "DC.to_csv(f'{data}/DrugCentral.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "移除下载文件:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stdout: \n"
     ]
    }
   ],
   "source": [
    "commands = f\"rm {data}/DC.tsv && rm {data}/DC_comps.tsv\"\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DTC\n",
    "\n",
    "下载约4.5h,建议在终端进行下载."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "commands = f\"wget --no-check-certificate https://drugtargetcommons.fimm.fi/static/Excell_files/DTC_data.csv -O {data}/DTC.csv\"\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, text=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## STITCH\n",
    "\n",
    "下载约5min."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "commands = f\"wget http://stitch.embl.de/download/protein_chemical.links.detailed.v5.0/9606.protein_chemical.links.detailed.v5.0.tsv.gz -O {data}/STITCH.tsv.gz && \\\n",
    "            gzip -d {data}/STITCH.tsv.gz\"\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, text=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Windows系统"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "commands = f\"dir {data}\"\n",
    "\n",
    "# Execute command\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BindingDB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BDB usually updates its database every month. The link includes the year and the month. \\\n",
    "If the link returns a 404 error, try manually updating the link with the previous month."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stdout: \n",
      "stderr: \n"
     ]
    }
   ],
   "source": [
    "m = datetime.now().strftime('%m')\n",
    "y = datetime.now().strftime('%Y') \n",
    "\n",
    "commands = f\"curl -o {data}\\\\BindingDB.tsv.zip https://www.bindingdb.org/bind/downloads/BindingDB_All_{y}{m}_tsv.zip && \\\n",
    "            tar -xf {data}\\\\BindingDB.tsv.zip -C {data}\\\\ && del {data}\\\\BindingDB.tsv.zip && \\\n",
    "                ren {data}\\\\BindingDB*.tsv BindingDB.tsv\"\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ChEMBL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ChEMBL database requires the generation of the `.csv` file from the [ChEMBL website](https://www.ebi.ac.uk/chembl/web_components/explore/activities/). In the page, select the *Homo sapiens* target organism in the filtering section on the right. Then, clicking on the csv download button will start the file generation. Once the generation is complete, press the download button and **save the file into the data folder**. The images below show the step by step procedure to download the file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Filtering section button for Homo sapiens](images/ChEMBL-filtering.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![CSV button to generate annotations file](images/ChEMBL-csv.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Button to download annotations file](images/ChEMBL-download.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The downloaded `.zip` file will contain multiple csv files that will need to be merged"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unzip file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stdout: \n",
      "stderr: \n"
     ]
    }
   ],
   "source": [
    "files = os.listdir(f\"{data}\\\\\")\n",
    "\n",
    "# # Downloaded ChEMBL file name will start with \"DOWNLOAD\"\n",
    "for file in files:\n",
    "    if file.startswith('DOWNLOAD') and file.endswith('.zip'):\n",
    "        zipped = file\n",
    "        break\n",
    "\n",
    "commands = f'mkdir {data}\\\\chembl && \\\n",
    "            tar -xf {data}\\\\{zipped} -C {data}\\\\chembl && \\\n",
    "            del \"{data}\\\\{zipped}\"'\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Merge files with the following code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def merge_csv_files(folder_path, output_file):\n",
    "    \n",
    "    files = os.listdir(folder_path)\n",
    "    \n",
    "    csv_files = [file for file in files if file.endswith('.csv')]\n",
    "    \n",
    "    if len(csv_files) == 0:\n",
    "        print(\"No CSV files found in the folder.\")\n",
    "        return\n",
    "    \n",
    "    merged_df = pd.DataFrame()\n",
    "    \n",
    "    for file in csv_files:\n",
    "        file_path = os.path.join(folder_path, file)\n",
    "        df = pd.read_csv(file_path, on_bad_lines='skip')\n",
    "        merged_df = pd.concat([merged_df, df], ignore_index=True)\n",
    "    \n",
    "    merged_df.to_csv(output_file, index=False)\n",
    "\n",
    "folder_path = f'{data}\\\\chembl'\n",
    "output_file = f'{data}\\\\ChEMBL.csv'\n",
    "merge_csv_files(folder_path, output_file)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove downloaded data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stdout: \n",
      "stderr: \n"
     ]
    }
   ],
   "source": [
    "commands = f\"rmdir /s /q {data}\\\\chembl\"\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CTD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stdout: \n",
      "stderr: \n"
     ]
    }
   ],
   "source": [
    "commands = f\"curl -L -o {data}\\\\CTD.csv.gz https://ctdbase.org/reports/CTD_chem_gene_ixns.csv.gz\"\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)\n",
    "\n",
    "# Extract csv using pandas\n",
    "ctd = pd.read_csv(f\"{data}\\\\CTD.csv.gz\", compression='gzip', on_bad_lines='skip', skiprows=28)\n",
    "ctd.to_csv(f\"{data}\\\\CTD.csv\", sep=',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove downloaded data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "commands = f\"del {data}\\\\CTD.csv.gz\"\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Drugbank"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The DrugBank database requires an academic license to download, please refer to the [DrugBank website](https://go.drugbank.com/releases/latest) for further instructions.\n",
    "Once the academic license has been issued to the account, please download the complete database from the [Drugbank download page](https://go.drugbank.com/releases/latest), or use the commands below. \\\n",
    "Then, unzip the file to obtain the `.xml` file. \n",
    "This file can be converted to `.csv` using the code provided below. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![DrugBank complete database download](images/Drugbank.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the zip file using your profile's data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stdout: \n",
      "stderr:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\n",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n",
      "100   477    0   477    0     0   1025      0 --:--:-- --:--:-- --:--:--  1030\n",
      "\n",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n",
      "  0  152M    0  8481    0     0   4936      0  8:58:29  0:00:01  8:58:28  9989\n",
      "  0  152M    0  433k    0     0   160k      0  0:16:11  0:00:02  0:16:09  236k\n",
      "  2  152M    2 4305k    0     0  1124k      0  0:02:18  0:00:03  0:02:15 1454k\n",
      "  5  152M    5 7873k    0     0  1630k      0  0:01:35  0:00:04  0:01:31 1988k\n",
      "  6  152M    6 10.4M    0     0  1874k      0  0:01:23  0:00:05  0:01:18 2212k\n",
      "  9  152M    9 13.7M    0     0  2091k      0  0:01:14  0:00:06  0:01:08 2808k\n",
      " 11  152M   11 17.2M    0     0  2288k      0  0:01:08  0:00:07  0:01:01 3438k\n",
      " 13  152M   13 20.7M    0     0  2442k      0  0:01:03  0:00:08  0:00:55 3479k\n",
      " 15  152M   15 24.2M    0     0  2562k      0  0:01:00  0:00:09  0:00:51 3486k\n",
      " 18  152M   18 27.9M    0     0  2676k      0  0:00:58  0:00:10  0:00:48 3589k\n",
      " 20  152M   20 31.6M    0     0  2774k      0  0:00:56  0:00:11  0:00:45 3696k\n",
      " 23  152M   23 35.4M    0     0  2858k      0  0:00:54  0:00:12  0:00:42 3738k\n",
      " 25  152M   25 38.9M    0     0  2916k      0  0:00:53  0:00:13  0:00:40 3740k\n",
      " 28  152M   28 42.5M    0     0  2934k      0  0:00:53  0:00:14  0:00:39 3631k\n",
      " 30  152M   30 45.6M    0     0  2981k      0  0:00:52  0:00:15  0:00:37 3635k\n",
      " 32  152M   32 48.7M    0     0  2991k      0  0:00:52  0:00:16  0:00:36 3500k\n",
      " 34  152M   34 52.1M    0     0  3017k      0  0:00:51  0:00:17  0:00:34 3419k\n",
      " 36  152M   36 55.7M    0     0  3055k      0  0:00:50  0:00:18  0:00:32 3436k\n",
      " 39  152M   39 59.4M    0     0  3090k      0  0:00:50  0:00:19  0:00:31 3570k\n",
      " 41  152M   41 63.2M    0     0  3130k      0  0:00:49  0:00:20  0:00:29 3599k\n",
      " 44  152M   44 67.0M    0     0  3136k      0  0:00:49  0:00:21  0:00:28 3599k\n",
      " 45  152M   45 69.8M    0     0  3154k      0  0:00:49  0:00:22  0:00:27 3638k\n",
      " 48  152M   48 73.0M    0     0  3156k      0  0:00:49  0:00:23  0:00:26 3535k\n",
      " 50  152M   50 76.4M    0     0  3165k      0  0:00:49  0:00:24  0:00:25 3460k\n",
      " 52  152M   52 79.9M    0     0  3184k      0  0:00:48  0:00:25  0:00:23 3405k\n",
      " 54  152M   54 83.5M    0     0  3206k      0  0:00:48  0:00:26  0:00:22 3529k\n",
      " 57  152M   57 87.1M    0     0  3221k      0  0:00:48  0:00:27  0:00:21 3526k\n",
      " 59  152M   59 90.9M    0     0  3244k      0  0:00:47  0:00:28  0:00:19 3662k\n",
      " 62  152M   62 94.7M    0     0  3267k      0  0:00:47  0:00:29  0:00:18 3777k\n",
      " 64  152M   64 98.4M    0     0  3262k      0  0:00:47  0:00:30  0:00:17 3646k\n",
      " 66  152M   66  101M    0     0  3269k      0  0:00:47  0:00:31  0:00:16 3603k\n",
      " 68  152M   68  104M    0     0  3267k      0  0:00:47  0:00:32  0:00:15 3523k\n",
      " 70  152M   70  107M    0     0  3275k      0  0:00:47  0:00:33  0:00:14 3452k\n",
      " 73  152M   73  111M    0     0  3287k      0  0:00:47  0:00:34  0:00:13 3401k\n",
      " 75  152M   75  114M    0     0  3292k      0  0:00:47  0:00:35  0:00:12 3490k\n",
      " 78  152M   78  118M    0     0  3310k      0  0:00:47  0:00:36  0:00:11 3566k\n",
      " 80  152M   80  122M    0     0  3325k      0  0:00:46  0:00:37  0:00:09 3702k\n",
      " 82  152M   82  126M    0     0  3339k      0  0:00:46  0:00:38  0:00:08 3773k\n",
      " 85  152M   85  129M    0     0  3333k      0  0:00:46  0:00:39  0:00:07 3657k\n",
      " 87  152M   87  132M    0     0  3330k      0  0:00:46  0:00:40  0:00:06 3604k\n",
      " 89  152M   89  135M    0     0  3334k      0  0:00:46  0:00:41  0:00:05 3512k\n",
      " 91  152M   91  139M    0     0  3341k      0  0:00:46  0:00:42  0:00:04 3463k\n",
      " 94  152M   94  143M    0     0  3352k      0  0:00:46  0:00:43  0:00:03 3451k\n",
      " 96  152M   96  146M    0     0  3364k      0  0:00:46  0:00:44  0:00:02 3604k\n",
      " 99  152M   99  150M    0     0  3375k      0  0:00:46  0:00:45  0:00:01 3735k\n",
      "100  152M  100  152M    0     0  3375k      0  0:00:46  0:00:46 --:--:-- 3758k\n",
      "\n"
     ]
    }
   ],
   "source": [
    "email = 'YOURUSERNAME'\n",
    "password = 'YOURPASSWORD'\n",
    "\n",
    "commands = f'curl -L --user {email}:{password} -o {data}\\\\drugbank_all_full_database.xml.zip https://go.drugbank.com/releases/latest/downloads/all-full-database'\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unzip the file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stdout: \n",
      "stderr: \n"
     ]
    }
   ],
   "source": [
    "commands = f\"tar -xf {data}\\\\drugbank_all_full_database.xml.zip -C {data}\\\\ && \\\n",
    "            del {data}\\\\drugbank_all_full_database.xml.zip\"\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def collapse_list_values(row):\n",
    "    for key, value in row.items():\n",
    "        if isinstance(value, list):\n",
    "            row[key] = '|'.join(value)\n",
    "    return row\n",
    "\n",
    "def xml2csv(file_path, output_file):\n",
    "    tree = ET.parse(file_path)\n",
    "    root = tree.getroot()\n",
    "\n",
    "    ns = '{http://www.drugbank.ca}'\n",
    "    inchikey_template = \"{ns}calculated-properties/{ns}property[{ns}kind='InChIKey']/{ns}value\"\n",
    "    inchi_template = \"{ns}calculated-properties/{ns}property[{ns}kind='InChI']/{ns}value\"\n",
    "\n",
    "    rows = list()\n",
    "    for i, drug in enumerate(root):\n",
    "        row = collections.OrderedDict()\n",
    "        assert drug.tag == ns + 'drug'\n",
    "        row['type'] = drug.get('type')\n",
    "        row['drugbank-id'] = drug.findtext(ns + \"drugbank-id[@primary='true']\")\n",
    "        row['name'] = drug.findtext(ns + \"name\")\n",
    "        row['description'] = drug.findtext(ns + \"description\")\n",
    "        row['InChIKey'] = drug.findtext(inchikey_template.format(ns = ns))\n",
    "        rows.append(row)\n",
    "\n",
    "    rows = list(map(collapse_list_values, rows))\n",
    "\n",
    "    columns = ['drugbank-id', 'name', 'type', 'InChIKey', 'description']\n",
    "    drugbank_df = pd.DataFrame.from_dict(rows)[columns]\n",
    "\n",
    "    protein_rows = list()\n",
    "    for i, drug in enumerate(root):\n",
    "        drugbank_id = drug.findtext(ns + \"drugbank-id[@primary='true']\")\n",
    "        for category in ['target', 'enzyme', 'carrier', 'transporter']:\n",
    "            proteins = drug.findall('{ns}{cat}s/{ns}{cat}'.format(ns=ns, cat=category))\n",
    "            for protein in proteins:\n",
    "                row = {'drugbank-id': drugbank_id, 'protein_type': category}\n",
    "                row['protein_name'] = protein.findtext('{}name'.format(ns))\n",
    "                row['organism'] = protein.findtext('{}organism'.format(ns))\n",
    "                actions = protein.findall('{ns}actions/{ns}action'.format(ns=ns))\n",
    "                row['actions'] = '|'.join(action.text for action in actions)\n",
    "                uniprot_ids = [polypep.text for polypep in protein.findall(\n",
    "                    \"{ns}polypeptide/{ns}external-identifiers/{ns}external-identifier[{ns}resource='UniProtKB']/{ns}identifier\".format(ns=ns))]            \n",
    "                if len(uniprot_ids) == 1:\n",
    "                    row['uniprot_id'] = uniprot_ids[0]\n",
    "                hgnc_ids = [polypep.text for polypep in protein.findall(\n",
    "                    \"{ns}polypeptide/{ns}external-identifiers/{ns}external-identifier[{ns}resource='HUGO Gene Nomenclature Committee (HGNC)']/{ns}identifier\".format(ns=ns))]            \n",
    "                if len(hgnc_ids) == 1:\n",
    "                    row['HGNC'] = hgnc_ids[0]\n",
    "                protein_rows.append(row)\n",
    "\n",
    "    protein_df = pd.DataFrame.from_dict(protein_rows)\n",
    "\n",
    "    drugbank = pd.merge(drugbank_df, protein_df, on='drugbank-id', how='left')\n",
    "\n",
    "    drugbank.to_csv(output_file, sep=',', index=False)\n",
    "\n",
    "file_path = f'{data}\\\\full database.xml'\n",
    "output_file = f'{data}\\\\DB.csv'\n",
    "xml2csv(file_path, output_file)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove downloaded data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stdout: \n",
      "stderr: \n"
     ]
    }
   ],
   "source": [
    "commands = f'del \"{data}\\\\full database.xml\"'\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DrugCentral"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stdout: \n",
      "stderr:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\n",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n",
      "  4  761k    4 31949    0     0  38012      0  0:00:20 --:--:--  0:00:20 38079\n",
      "100  761k  100  761k    0     0   581k      0  0:00:01  0:00:01 --:--:--  582k\n",
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\n",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n",
      "100 1063k  100 1063k    0     0   754k      0  0:00:01  0:00:01 --:--:--  756k\n",
      "\n"
     ]
    }
   ],
   "source": [
    "commands = f\"curl -o {data}\\\\DC.tsv.gz https://unmtid-dbs.net/download/DrugCentral/2021_09_01/drug.target.interaction.tsv.gz && \\\n",
    "            curl -o {data}\\\\DC_comps.tsv https://unmtid-dbs.net/download/DrugCentral/2021_09_01/structures.smiles.tsv\"\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Merge the two files to add SMILES and InChI information for compounds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "DC = pd.read_csv(f'{data}\\\\DC.tsv.gz', sep='\\t', compression='gzip', on_bad_lines='skip')\n",
    "DC_comps = pd.read_csv(f'{data}\\\\DC_comps.tsv', sep='\\t')\n",
    "\n",
    "DC_comps = DC_comps[['ID', 'SMILES', 'InChI', 'InChIKey']].rename(columns={'ID':'STRUCT_ID'})\n",
    "\n",
    "DC = pd.merge(DC, DC_comps, on='STRUCT_ID', how='left')\n",
    "\n",
    "DC.to_csv(f'{data}\\\\DrugCentral.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove downloaded data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stdout: \n",
      "stderr: \n"
     ]
    }
   ],
   "source": [
    "commands = f\"del {data}\\\\DC.tsv.gz && del {data}\\\\DC_comps.tsv\"\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DTC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stdout: \n",
      "stderr:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\n",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n",
      "  0 2168M    0  135k    0     0  80468      0  7:50:57  0:00:01  7:50:56 80532\n",
      "  0 2168M    0 1223k    0     0   460k      0  1:20:23  0:00:02  1:20:21  460k\n",
      "  1 2168M    1 24.8M    0     0  6966k      0  0:05:18  0:00:03  0:05:15 6969k\n",
      "  2 2168M    2 53.4M    0     0  11.4M      0  0:03:08  0:00:04  0:03:04 11.4M\n",
      "  3 2168M    3 78.5M    0     0  13.8M      0  0:02:36  0:00:05  0:02:31 15.8M\n",
      "  4 2168M    4  106M    0     0  15.9M      0  0:02:15  0:00:06  0:02:09 21.5M\n",
      "  6 2168M    6  134M    0     0  17.6M      0  0:02:03  0:00:07  0:01:56 26.7M\n",
      "  7 2168M    7  165M    0     0  19.1M      0  0:01:53  0:00:08  0:01:45 28.2M\n",
      "  8 2168M    8  190M    0     0  19.6M      0  0:01:50  0:00:09  0:01:41 27.3M\n",
      " 10 2168M   10  220M    0     0  20.6M      0  0:01:44  0:00:10  0:01:34 28.3M\n",
      " 11 2168M   11  247M    0     0  21.2M      0  0:01:41  0:00:11  0:01:30 28.3M\n",
      " 12 2168M   12  277M    0     0  21.8M      0  0:01:39  0:00:12  0:01:27 28.4M\n",
      " 13 2168M   13  301M    0     0  22.0M      0  0:01:38  0:00:13  0:01:25 27.0M\n",
      " 15 2168M   15  332M    0     0  22.6M      0  0:01:35  0:00:14  0:01:21 28.4M\n",
      " 16 2168M   16  361M    0     0  23.0M      0  0:01:33  0:00:15  0:01:18 28.3M\n",
      " 17 2168M   17  388M    0     0  23.2M      0  0:01:33  0:00:16  0:01:17 27.9M\n",
      " 19 2168M   19  414M    0     0  23.4M      0  0:01:32  0:00:17  0:01:15 27.4M\n",
      " 20 2168M   20  442M    0     0  23.7M      0  0:01:31  0:00:18  0:01:13 28.2M\n",
      " 20 2168M   20  450M    0     0  22.9M      0  0:01:34  0:00:19  0:01:15 23.6M\n",
      " 21 2168M   21  464M    0     0  22.4M      0  0:01:36  0:00:20  0:01:16 20.5M\n",
      " 22 2168M   22  490M    0     0  22.6M      0  0:01:35  0:00:21  0:01:14 20.6M\n",
      " 23 2168M   23  518M    0     0  22.8M      0  0:01:34  0:00:22  0:01:12 20.8M\n",
      " 25 2168M   25  548M    0     0  23.1M      0  0:01:33  0:00:23  0:01:10 21.2M\n",
      " 26 2168M   26  576M    0     0  23.3M      0  0:01:32  0:00:24  0:01:08 24.9M\n",
      " 27 2168M   27  603M    0     0  23.5M      0  0:01:32  0:00:25  0:01:07 27.7M\n",
      " 29 2168M   29  631M    0     0  23.6M      0  0:01:31  0:00:26  0:01:05 28.0M\n",
      " 30 2168M   30  657M    0     0  23.7M      0  0:01:31  0:00:27  0:01:04 27.6M\n",
      " 31 2168M   31  686M    0     0  23.9M      0  0:01:30  0:00:28  0:01:02 27.5M\n",
      " 32 2168M   32  713M    0     0  24.0M      0  0:01:30  0:00:29  0:01:01 27.5M\n",
      " 34 2168M   34  741M    0     0  24.1M      0  0:01:29  0:00:30  0:00:59 27.6M\n",
      " 35 2168M   35  768M    0     0  24.2M      0  0:01:29  0:00:31  0:00:58 27.4M\n",
      " 36 2168M   36  799M    0     0  24.0M      0  0:01:30  0:00:33  0:00:57 25.3M\n",
      " 37 2168M   37  818M    0     0  24.3M      0  0:01:29  0:00:33  0:00:56 26.5M\n",
      " 39 2168M   39  847M    0     0  24.4M      0  0:01:28  0:00:34  0:00:54 26.7M\n",
      " 39 2168M   39  852M    0     0  23.7M      0  0:01:31  0:00:35  0:00:56 21.3M\n",
      " 39 2168M   39  852M    0     0  23.1M      0  0:01:33  0:00:36  0:00:57 16.2M\n",
      " 39 2168M   39  852M    0     0  22.5M      0  0:01:36  0:00:37  0:00:59 11.5M\n",
      " 39 2168M   39  852M    0     0  22.0M      0  0:01:38  0:00:38  0:01:00 6833k\n",
      " 39 2168M   39  852M    0     0  21.4M      0  0:01:40  0:00:39  0:01:01 1095k\n",
      " 39 2168M   39  852M    0     0  20.9M      0  0:01:43  0:00:40  0:01:03 67203\n",
      " 39 2168M   39  852M    0     0  20.3M      0  0:01:46  0:00:42  0:01:04 82095\n",
      " 39 2168M   39  852M    0     0  19.8M      0  0:01:49  0:00:43  0:01:06 82223\n",
      " 39 2168M   39  852M    0     0  19.3M      0  0:01:51  0:00:44  0:01:07 73912\n",
      " 39 2168M   39  852M    0     0  18.9M      0  0:01:54  0:00:45  0:01:09 50548\n",
      " 39 2168M   39  852M    0     0  18.5M      0  0:01:57  0:00:46  0:01:11 18381\n",
      " 39 2168M   39  852M    0     0  18.1M      0  0:01:59  0:00:47  0:01:12     0\n",
      " 39 2168M   39  852M    0     0  17.8M      0  0:02:01  0:00:47  0:01:14   845\n",
      " 39 2168M   39  852M    0     0  17.5M      0  0:02:03  0:00:48  0:01:15  5277\n",
      " 39 2168M   39  852M    0     0  17.1M      0  0:02:06  0:00:49  0:01:17 11226\n",
      " 39 2168M   39  852M    0     0  16.8M      0  0:02:08  0:00:50  0:01:18 14045\n",
      " 39 2168M   39  853M    0     0  16.5M      0  0:02:11  0:00:51  0:01:20 64706\n",
      " 39 2168M   39  853M    0     0  16.2M      0  0:02:13  0:00:52  0:01:21  208k\n",
      " 39 2168M   39  855M    0     0  15.9M      0  0:02:16  0:00:53  0:01:23  442k\n",
      " 39 2168M   39  856M    0     0  15.6M      0  0:02:18  0:00:54  0:01:24  761k\n",
      " 39 2168M   39  858M    0     0  15.4M      0  0:02:20  0:00:55  0:01:25 1176k\n",
      " 39 2168M   39  861M    0     0  15.1M      0  0:02:22  0:00:56  0:01:26 1626k\n",
      " 39 2168M   39  863M    0     0  14.9M      0  0:02:24  0:00:57  0:01:27 2033k\n",
      " 39 2168M   39  867M    0     0  14.7M      0  0:02:26  0:00:58  0:01:28 2457k\n",
      " 40 2168M   40  870M    0     0  14.5M      0  0:02:28  0:00:59  0:01:29 2885k\n",
      " 40 2168M   40  872M    0     0  14.3M      0  0:02:30  0:01:00  0:01:30 2951k\n",
      " 40 2168M   40  875M    0     0  14.2M      0  0:02:32  0:01:01  0:01:31 2970k\n",
      " 40 2168M   40  878M    0     0  14.0M      0  0:02:34  0:01:02  0:01:32 3030k\n",
      " 40 2168M   40  881M    0     0  13.8M      0  0:02:36  0:01:03  0:01:33 3042k\n",
      " 40 2168M   40  885M    0     0  13.6M      0  0:02:38  0:01:04  0:01:34 3077k\n",
      " 41 2168M   41  889M    0     0  13.5M      0  0:02:40  0:01:05  0:01:35 3446k\n",
      " 41 2168M   41  894M    0     0  13.4M      0  0:02:41  0:01:06  0:01:35 3862k\n",
      " 41 2168M   41  899M    0     0  13.2M      0  0:02:43  0:01:07  0:01:36 4240k\n",
      " 41 2168M   41  904M    0     0  13.1M      0  0:02:44  0:01:08  0:01:36 4616k\n",
      " 41 2168M   41  910M    0     0  13.0M      0  0:02:45  0:01:09  0:01:36 5000k\n",
      " 42 2168M   42  915M    0     0  12.9M      0  0:02:47  0:01:10  0:01:37 5306k\n",
      " 42 2168M   42  921M    0     0  12.8M      0  0:02:48  0:01:11  0:01:37 5619k\n",
      " 42 2168M   42  928M    0     0  12.7M      0  0:02:49  0:01:12  0:01:37 5974k\n",
      " 43 2168M   43  935M    0     0  12.6M      0  0:02:50  0:01:13  0:01:37 6324k\n",
      " 43 2168M   43  939M    0     0  12.5M      0  0:02:52  0:01:14  0:01:38 5983k\n",
      " 43 2168M   43  943M    0     0  12.4M      0  0:02:53  0:01:15  0:01:38 5642k\n",
      " 43 2168M   43  947M    0     0  12.3M      0  0:02:55  0:01:16  0:01:39 5259k\n",
      " 43 2168M   43  952M    0     0  12.2M      0  0:02:56  0:01:17  0:01:39 4856k\n",
      " 44 2168M   44  957M    0     0  12.1M      0  0:02:58  0:01:18  0:01:40 4481k\n",
      " 44 2168M   44  962M    0     0  12.0M      0  0:02:59  0:01:19  0:01:40 4767k\n",
      " 44 2168M   44  968M    0     0  12.0M      0  0:03:00  0:01:20  0:01:40 5074k\n",
      " 44 2168M   44  974M    0     0  11.9M      0  0:03:01  0:01:21  0:01:40 5439k\n",
      " 45 2168M   45  980M    0     0  11.8M      0  0:03:02  0:01:22  0:01:40 5780k\n",
      " 45 2168M   45  986M    0     0  11.7M      0  0:03:03  0:01:23  0:01:40 6051k\n",
      " 45 2168M   45  993M    0     0  11.7M      0  0:03:04  0:01:24  0:01:40 6342k\n",
      " 46 2168M   46 1000M    0     0  11.6M      0  0:03:05  0:01:25  0:01:40 6644k\n",
      " 46 2168M   46 1007M    0     0  11.6M      0  0:03:06  0:01:26  0:01:40 6891k\n",
      " 46 2168M   46 1015M    0     0  11.5M      0  0:03:07  0:01:27  0:01:40 7171k\n",
      " 47 2168M   47 1020M    0     0  11.5M      0  0:03:08  0:01:28  0:01:40 7000k\n",
      " 47 2168M   47 1025M    0     0  11.4M      0  0:03:09  0:01:29  0:01:40 6505k\n",
      " 47 2168M   47 1030M    0     0  11.3M      0  0:03:10  0:01:30  0:01:40 6040k\n",
      " 47 2168M   47 1035M    0     0  11.2M      0  0:03:12  0:01:31  0:01:41 5598k\n",
      " 47 2168M   47 1040M    0     0  11.2M      0  0:03:13  0:01:32  0:01:41 5150k\n",
      " 48 2168M   48 1046M    0     0  11.1M      0  0:03:14  0:01:33  0:01:41 5159k\n",
      " 48 2168M   48 1051M    0     0  11.1M      0  0:03:15  0:01:34  0:01:41 5448k\n",
      " 48 2168M   48 1058M    0     0  11.0M      0  0:03:16  0:01:35  0:01:41 5741k\n",
      " 49 2168M   49 1064M    0     0  11.0M      0  0:03:16  0:01:36  0:01:40 6007k\n",
      " 49 2168M   49 1070M    0     0  10.9M      0  0:03:17  0:01:37  0:01:40 6228k\n",
      " 49 2168M   49 1077M    0     0  10.9M      0  0:03:18  0:01:38  0:01:40 6503k\n",
      " 50 2168M   50 1084M    0     0  10.8M      0  0:03:19  0:01:39  0:01:40 6763k\n",
      " 50 2168M   50 1092M    0     0  10.8M      0  0:03:19  0:01:40  0:01:39 7019k\n",
      " 50 2168M   50 1100M    0     0  10.8M      0  0:03:20  0:01:41  0:01:39 7289k\n",
      " 51 2168M   51 1107M    0     0  10.7M      0  0:03:20  0:01:42  0:01:38 7537k\n",
      " 51 2168M   51 1115M    0     0  10.7M      0  0:03:21  0:01:43  0:01:38 7681k\n",
      " 51 2168M   51 1123M    0     0  10.7M      0  0:03:22  0:01:44  0:01:38 7809k\n",
      " 52 2168M   52 1131M    0     0  10.7M      0  0:03:22  0:01:45  0:01:37 8040k\n",
      " 52 2168M   52 1139M    0     0  10.6M      0  0:03:22  0:01:46  0:01:36 8120k\n",
      " 52 2168M   52 1148M    0     0  10.6M      0  0:03:23  0:01:47  0:01:36 8331k\n",
      " 53 2168M   53 1157M    0     0  10.6M      0  0:03:23  0:01:48  0:01:35 8554k\n",
      " 53 2168M   53 1165M    0     0  10.6M      0  0:03:23  0:01:49  0:01:34 8761k\n",
      " 54 2168M   54 1174M    0     0  10.6M      0  0:03:24  0:01:50  0:01:34 8804k\n",
      " 54 2168M   54 1183M    0     0  10.5M      0  0:03:24  0:01:51  0:01:33 8984k\n",
      " 55 2168M   55 1192M    0     0  10.5M      0  0:03:24  0:01:52  0:01:32 9071k\n",
      " 55 2168M   55 1202M    0     0  10.5M      0  0:03:25  0:01:53  0:01:32 9214k\n",
      " 55 2168M   55 1211M    0     0  10.5M      0  0:03:25  0:01:54  0:01:31 9357k\n",
      " 56 2168M   56 1221M    0     0  10.5M      0  0:03:25  0:01:55  0:01:30 9567k\n",
      " 56 2168M   56 1230M    0     0  10.5M      0  0:03:25  0:01:56  0:01:29 9684k\n",
      " 57 2168M   57 1241M    0     0  10.5M      0  0:03:25  0:01:57  0:01:28 9901k\n",
      " 57 2168M   57 1251M    0     0  10.5M      0  0:03:25  0:01:58  0:01:27  9.7M\n",
      " 58 2168M   58 1261M    0     0  10.5M      0  0:03:25  0:01:59  0:01:26  9.9M\n",
      " 58 2168M   58 1271M    0     0  10.5M      0  0:03:25  0:02:00  0:01:25 10.1M\n",
      " 59 2168M   59 1281M    0     0  10.5M      0  0:03:25  0:02:01  0:01:24 10.2M\n",
      " 59 2168M   59 1292M    0     0  10.5M      0  0:03:25  0:02:02  0:01:23 10.2M\n",
      " 60 2168M   60 1302M    0     0  10.5M      0  0:03:25  0:02:03  0:01:22 10.3M\n",
      " 60 2168M   60 1313M    0     0  10.5M      0  0:03:25  0:02:04  0:01:21 10.5M\n",
      " 61 2168M   61 1324M    0     0  10.5M      0  0:03:25  0:02:05  0:01:20 10.5M\n",
      " 61 2168M   61 1335M    0     0  10.5M      0  0:03:25  0:02:06  0:01:19 10.7M\n",
      " 62 2168M   62 1346M    0     0  10.5M      0  0:03:25  0:02:07  0:01:18 10.7M\n",
      " 62 2168M   62 1357M    0     0  10.5M      0  0:03:25  0:02:08  0:01:17 10.9M\n",
      " 63 2168M   63 1368M    0     0  10.5M      0  0:03:25  0:02:09  0:01:16 10.9M\n",
      " 63 2168M   63 1380M    0     0  10.5M      0  0:03:25  0:02:10  0:01:15 11.1M\n",
      " 64 2168M   64 1391M    0     0  10.5M      0  0:03:25  0:02:11  0:01:14 11.2M\n",
      " 64 2168M   64 1403M    0     0  10.5M      0  0:03:24  0:02:12  0:01:12 11.3M\n",
      " 65 2168M   65 1415M    0     0  10.5M      0  0:03:24  0:02:13  0:01:11 11.5M\n",
      " 65 2168M   65 1426M    0     0  10.5M      0  0:03:24  0:02:14  0:01:10 11.6M\n",
      " 66 2168M   66 1436M    0     0  10.5M      0  0:03:24  0:02:15  0:01:09 11.3M\n",
      " 66 2168M   66 1441M    0     0  10.5M      0  0:03:25  0:02:16  0:01:09  9.8M\n",
      " 66 2168M   66 1444M    0     0  10.4M      0  0:03:26  0:02:17  0:01:09 8502k\n",
      " 66 2168M   66 1448M    0     0  10.4M      0  0:03:27  0:02:18  0:01:09 6900k\n",
      " 67 2168M   67 1453M    0     0  10.4M      0  0:03:28  0:02:19  0:01:09 5443k\n",
      " 67 2168M   67 1458M    0     0  10.3M      0  0:03:29  0:02:20  0:01:09 4370k\n",
      " 67 2168M   67 1463M    0     0  10.3M      0  0:03:29  0:02:21  0:01:08 4514k\n",
      " 67 2168M   67 1468M    0     0  10.2M      0  0:03:30  0:02:22  0:01:08 4867k\n",
      " 67 2168M   67 1474M    0     0  10.2M      0  0:03:31  0:02:23  0:01:08 5171k\n",
      " 68 2168M   68 1480M    0     0  10.2M      0  0:03:31  0:02:24  0:01:07 5519k\n",
      " 68 2168M   68 1486M    0     0  10.2M      0  0:03:32  0:02:25  0:01:07 5863k\n",
      " 68 2168M   68 1493M    0     0  10.1M      0  0:03:32  0:02:26  0:01:06 6266k\n",
      " 69 2168M   69 1500M    0     0  10.1M      0  0:03:33  0:02:27  0:01:06 6624k\n",
      " 69 2168M   69 1508M    0     0  10.1M      0  0:03:33  0:02:28  0:01:05 6936k\n",
      " 69 2168M   69 1515M    0     0  10.1M      0  0:03:34  0:02:29  0:01:05 7136k\n",
      " 70 2168M   70 1521M    0     0  10.0M      0  0:03:34  0:02:30  0:01:04 7097k\n",
      " 70 2168M   70 1524M    0     0  10.0M      0  0:03:35  0:02:31  0:01:04 6380k\n",
      " 70 2168M   70 1527M    0     0  10.0M      0  0:03:36  0:02:32  0:01:04 5438k\n",
      " 70 2168M   70 1530M    0     0   9.9M      0  0:03:37  0:02:33  0:01:04 4609k\n",
      " 70 2168M   70 1533M    0     0   9.9M      0  0:03:38  0:02:34  0:01:04 3845k\n",
      " 70 2168M   70 1537M    0     0   9.8M      0  0:03:39  0:02:35  0:01:04 3318k\n",
      " 71 2168M   71 1541M    0     0   9.8M      0  0:03:40  0:02:36  0:01:04 3485k\n",
      " 71 2168M   71 1546M    0     0   9.8M      0  0:03:41  0:02:37  0:01:04 3865k\n",
      " 71 2168M   71 1551M    0     0   9.7M      0  0:03:41  0:02:38  0:01:03 4203k\n",
      " 71 2168M   71 1556M    0     0  9981k      0  0:03:42  0:02:39  0:01:03 4559k\n",
      " 72 2168M   72 1561M    0     0  9953k      0  0:03:43  0:02:40  0:01:03 4894k\n",
      " 72 2168M   72 1567M    0     0  9927k      0  0:03:43  0:02:41  0:01:02 5182k\n",
      " 72 2168M   72 1573M    0     0  9904k      0  0:03:44  0:02:42  0:01:02 5498k\n",
      " 72 2168M   72 1579M    0     0  9883k      0  0:03:44  0:02:43  0:01:01 5844k\n",
      " 73 2168M   73 1586M    0     0  9864k      0  0:03:45  0:02:44  0:01:01 6131k\n",
      " 73 2168M   73 1592M    0     0  9846k      0  0:03:45  0:02:45  0:01:00 6426k\n",
      " 73 2168M   73 1599M    0     0  9830k      0  0:03:45  0:02:46  0:00:59 6697k\n",
      " 74 2168M   74 1607M    0     0  9816k      0  0:03:46  0:02:47  0:00:59 6951k\n",
      " 74 2168M   74 1614M    0     0  9802k      0  0:03:46  0:02:48  0:00:58 7148k\n",
      " 74 2168M   74 1622M    0     0  9790k      0  0:03:46  0:02:49  0:00:57 7356k\n",
      " 75 2168M   75 1629M    0     0  9776k      0  0:03:47  0:02:50  0:00:57 7460k\n",
      " 75 2168M   75 1637M    0     0  9767k      0  0:03:47  0:02:51  0:00:56 7675k\n",
      " 75 2168M   75 1645M    0     0  9759k      0  0:03:47  0:02:52  0:00:55 7863k\n",
      " 76 2168M   76 1653M    0     0  9751k      0  0:03:47  0:02:53  0:00:54 8041k\n",
      " 76 2168M   76 1662M    0     0  9747k      0  0:03:47  0:02:54  0:00:53 8294k\n",
      " 77 2168M   77 1671M    0     0  9743k      0  0:03:47  0:02:55  0:00:52 8590k\n",
      " 77 2168M   77 1680M    0     0  9739k      0  0:03:47  0:02:56  0:00:51 8793k\n",
      " 77 2168M   77 1689M    0     0  9737k      0  0:03:48  0:02:57  0:00:51 8987k\n",
      " 78 2168M   78 1698M    0     0  9737k      0  0:03:48  0:02:58  0:00:50 9242k\n",
      " 78 2168M   78 1708M    0     0  9738k      0  0:03:48  0:02:59  0:00:49 9412k\n",
      " 79 2168M   79 1718M    0     0  9738k      0  0:03:48  0:03:00  0:00:48 9566k\n",
      " 79 2168M   79 1727M    0     0  9737k      0  0:03:48  0:03:01  0:00:47 9655k\n",
      " 80 2168M   80 1737M    0     0  9738k      0  0:03:48  0:03:02  0:00:46 9754k\n",
      " 80 2168M   80 1747M    0     0  9741k      0  0:03:47  0:03:03  0:00:44 9894k\n",
      " 81 2168M   81 1757M    0     0  9744k      0  0:03:47  0:03:04  0:00:43 9943k\n",
      " 81 2168M   81 1767M    0     0  9748k      0  0:03:47  0:03:05  0:00:42  9.8M\n",
      " 81 2168M   81 1777M    0     0  9750k      0  0:03:47  0:03:06  0:00:41  9.9M\n",
      " 82 2168M   82 1788M    0     0  9758k      0  0:03:47  0:03:07  0:00:40 10.2M\n",
      " 82 2168M   82 1798M    0     0  9762k      0  0:03:47  0:03:08  0:00:39 10.2M\n",
      " 83 2168M   83 1809M    0     0  9770k      0  0:03:47  0:03:09  0:00:38 10.5M\n",
      " 83 2168M   83 1820M    0     0  9776k      0  0:03:47  0:03:10  0:00:37 10.5M\n",
      " 84 2168M   84 1831M    0     0  9784k      0  0:03:46  0:03:11  0:00:35 10.7M\n",
      " 84 2168M   84 1842M    0     0  9792k      0  0:03:46  0:03:12  0:00:34 10.8M\n",
      " 85 2168M   85 1853M    0     0  9801k      0  0:03:46  0:03:13  0:00:33 10.9M\n",
      " 85 2168M   85 1861M    0     0  9793k      0  0:03:46  0:03:14  0:00:32 10.4M\n",
      " 86 2168M   86 1867M    0     0  9776k      0  0:03:47  0:03:15  0:00:32 9753k\n",
      " 86 2168M   86 1874M    0     0  9758k      0  0:03:47  0:03:16  0:00:31 8783k\n",
      " 86 2168M   86 1880M    0     0  9743k      0  0:03:47  0:03:17  0:00:30 7843k\n",
      " 87 2168M   87 1887M    0     0  9727k      0  0:03:48  0:03:18  0:00:30 6857k\n",
      " 87 2168M   87 1893M    0     0  9713k      0  0:03:48  0:03:19  0:00:29 6578k\n",
      " 87 2168M   87 1900M    0     0  9700k      0  0:03:48  0:03:20  0:00:28 6741k\n",
      " 88 2168M   88 1908M    0     0  9690k      0  0:03:49  0:03:21  0:00:28 7002k\n",
      " 88 2168M   88 1915M    0     0  9679k      0  0:03:49  0:03:22  0:00:27 7138k\n",
      " 88 2168M   88 1921M    0     0  9659k      0  0:03:49  0:03:23  0:00:26 6983k\n",
      " 88 2168M   88 1925M    0     0  9633k      0  0:03:50  0:03:24  0:00:26 6460k\n",
      " 88 2168M   88 1929M    0     0  9608k      0  0:03:51  0:03:25  0:00:26 5901k\n",
      " 89 2168M   89 1934M    0     0  9585k      0  0:03:51  0:03:26  0:00:25 5347k\n",
      " 89 2168M   89 1939M    0     0  9564k      0  0:03:52  0:03:27  0:00:25 4899k\n",
      " 89 2168M   89 1945M    0     0  9545k      0  0:03:52  0:03:28  0:00:24 4891k\n",
      " 89 2168M   89 1950M    0     0  9528k      0  0:03:53  0:03:29  0:00:24 5218k\n",
      " 90 2168M   90 1956M    0     0  9512k      0  0:03:53  0:03:30  0:00:23 5574k\n",
      " 90 2168M   90 1963M    0     0  9497k      0  0:03:53  0:03:31  0:00:22 5882k\n",
      " 90 2168M   90 1969M    0     0  9484k      0  0:03:54  0:03:32  0:00:22 6172k\n",
      " 91 2168M   91 1976M    0     0  9472k      0  0:03:54  0:03:33  0:00:21 6417k\n",
      " 91 2168M   91 1983M    0     0  9460k      0  0:03:54  0:03:34  0:00:20 6620k\n",
      " 91 2168M   91 1990M    0     0  9449k      0  0:03:54  0:03:35  0:00:19 6808k\n",
      " 92 2168M   92 1997M    0     0  9440k      0  0:03:55  0:03:36  0:00:19 7013k\n",
      " 92 2168M   92 2004M    0     0  9432k      0  0:03:55  0:03:37  0:00:18 7213k\n",
      " 92 2168M   92 2012M    0     0  9424k      0  0:03:55  0:03:38  0:00:17 7401k\n",
      " 93 2168M   93 2020M    0     0  9419k      0  0:03:55  0:03:39  0:00:16 7646k\n",
      " 93 2168M   93 2028M    0     0  9413k      0  0:03:55  0:03:40  0:00:15 7862k\n",
      " 93 2168M   93 2036M    0     0  9409k      0  0:03:55  0:03:41  0:00:14 8068k\n",
      " 94 2168M   94 2044M    0     0  9404k      0  0:03:56  0:03:42  0:00:14 8197k\n",
      " 94 2168M   94 2053M    0     0  9400k      0  0:03:56  0:03:43  0:00:13 8335k\n",
      " 95 2168M   95 2062M    0     0  9399k      0  0:03:56  0:03:44  0:00:12 8517k\n",
      " 95 2168M   95 2071M    0     0  9398k      0  0:03:56  0:03:45  0:00:11 8733k\n",
      " 95 2168M   95 2080M    0     0  9399k      0  0:03:56  0:03:46  0:00:10 8937k\n",
      " 96 2168M   96 2089M    0     0  9398k      0  0:03:56  0:03:47  0:00:09 9129k\n",
      " 96 2168M   96 2098M    0     0  9397k      0  0:03:56  0:03:48  0:00:08 9282k\n",
      " 97 2168M   97 2108M    0     0  9399k      0  0:03:56  0:03:49  0:00:07 9423k\n",
      " 97 2168M   97 2117M    0     0  9401k      0  0:03:56  0:03:50  0:00:06 9520k\n",
      " 98 2168M   98 2127M    0     0  9402k      0  0:03:56  0:03:51  0:00:05 9536k\n",
      " 98 2168M   98 2136M    0     0  9402k      0  0:03:56  0:03:52  0:00:04 9589k\n",
      " 98 2168M   98 2146M    0     0  9404k      0  0:03:56  0:03:53  0:00:03 9713k\n",
      " 99 2168M   99 2155M    0     0  9408k      0  0:03:56  0:03:54  0:00:02 9792k\n",
      " 99 2168M   99 2165M    0     0  9411k      0  0:03:55  0:03:55 --:--:-- 9863k\n",
      "100 2168M  100 2168M    0     0  9412k      0  0:03:55  0:03:55 --:--:-- 9984k\n",
      "\n"
     ]
    }
   ],
   "source": [
    "commands = f\"curl -o {data}\\\\DTC.csv https://drugtargetcommons.fimm.fi/static/Excell_files/DTC_data.csv\"\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## STITCH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "commands = f\"curl -o {data}\\\\STITCH.tsv.gz http://stitch.embl.de/download/protein_chemical.links.detailed.v5.0/9606.protein_chemical.links.detailed.v5.0.tsv.gz\"\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)\n",
    "\n",
    "# Extract csv using pandas\n",
    "stitch = pd.read_csv(f\"{data}\\\\STICH.tsv.gz\", sep='\\t', compression='gzip', on_bad_lines='skip')\n",
    "stitch.to_csv(f\"{data}\\\\STITCH.tsv\", sep='\\t')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Delete downloaded data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stdout: \n",
      "stderr: \n"
     ]
    }
   ],
   "source": [
    "commands = f\"del {data}\\\\STITCH.tsv.gz\"\n",
    "\n",
    "# Execute commands\n",
    "result = subprocess.run(commands, shell=True, capture_output=True, text=True)\n",
    "\n",
    "# Print the standard output\n",
    "print('stdout:', result.stdout)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "cpi",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
